View Synthesis for Multiview Video Compression

We consider multiview video compression: the problem of jointly compressing multiple views of a scene recorded by different cameras. To take advantage of the correlation between views, we compare the performance of disparity compensated view prediction and view synthesis prediction to independent coding of all views using H.264/AVC. The proposed view synthesis prediction technique works by first synthesizing a virtual version of each view using previously encoded views and using the virtual view as a reference for predictive coding. We present experimental coding results showing that view synthesis prediction has the potential to perform significantly better than both disparity compensated view prediction and independent coding of all views. Accepted for publication in Picture Coding Symposium 2006 This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Research Laboratories, Inc.; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Research Laboratories, Inc. All rights reserved. Copyright c © Mitsubishi Electric Research Laboratories, Inc., 2006 201 Broadway, Cambridge, Massachusetts 02139 Publication History:– 1. First printing, TR2006-035, April 2006 View Synthesis for Multiview Video Compression Emin Martinian, Alexander Behrens, Jun Xin, and Anthony Vetro email:{martinian,jxin,avetro}@merl.com, [email protected] Mitsubishi Electric Research Labs 201 Broadway Cambridge, MA 02139, USA Abstract. We consider multiview video compression: the problem of jointly compressing multiple views of a scene recorded by different cameras. To take advantage We consider multiview video compression: the problem of jointly compressing multiple views of a scene recorded by different cameras. To take advantage of the correlation between views, we compare the performance of disparity compensated view prediction and view synthesis prediction to independent coding of all views using H.264/AVC. The proposed view synthesis prediction technique works by first synthesizing a virtual version of each view using previously encoded views and using the virtual view as a reference for predictive coding. We present experimental coding results showing that view synthesis prediction has the potential to perform significantly better than both disparity compensated view prediction and independent coding of all views. Index Terms view synthesis, view interpolation, multiview video compression, H.264/AVC